Product modeling system and method

ABSTRACT

A product modeling system and method are provided. On one embodiment, the product modeling system is used to model a piece of apparel, such as a shirt, with a design wherein the model with the design is used to display the piece of apparel with the design to a consumer.

PRIORITY CLAIMS/RELATED APPLICATIONS

This application is a continuation of and claims priority under 35 USC120 to U.S. patent application Ser. No. 13/968,142 filed Aug. 15, 2013which is a continuation of and claims priority under 35 USC 120 to U.S.patent application Ser. No. 13/464,551 filed on May 4, 2012 (to beissued on Aug. 20, 2013 as U.S. Pat. No. 8,514,220) which application inturn is a continuation of and claims priority under 35 USC 120 to U.S.patent application Ser. No. 11/925,716 filed on Oct. 26, 2007 (issued onMay 8, 2012 as U.S. Pat. No. 8,174,521), the entirety of which isincorporated herein by reference.

APPENDICES

Appendix A (2 pages) contains an example of the pseudocode for finding aset of markers on a product;

Appendix B (1 page) contains an example of the code for remapping thecolor of an image using normalized ordinal color distribution;

Appendix C (4 pages) contains an example of the code for building acolor map in normalized histogram order with an index from a matchingcolor space;

Appendix D (2 pages) contains an example of the code for building alook-up table to remap the colors from a source sphere to a destinationsphere; and

Appendix E (3 pages) contains an example of the code for remapping thecolor of the source image with a source sphere color map to adestination image with the color map of the sphere color object.

Appendices A-E form part of the specification and are incorporatedherein by reference.

FIELD

The invention relates generally to a system and method for modeling apiece of apparel.

BACKGROUND

Electronic commerce (E-commerce) is a thriving business in which variousdifferent products and services are sold to a plurality of consumersusing an E-commerce site. The E-Commerce site may include a website thatallows a plurality of consumers to gain access to the website using anetwork, such as the Internet. The website may have a plurality of webpages wherein these web pages have images of a plurality of differentproducts that the consumer may purchase. The images contained in theplurality of web pages are two dimensional images. The website may alsoinclude a secure commerce portion that allows the consumer to select oneor more items, place those items in an electronic shopping cart and,when done shopping, check out and pay for the items that remain in theelectronic shopping cart using various payment services, such as PayPalor a credit card.

One limitation with these typical E-commerce systems is that the productavailable on the website, such as a shirt, may be modeled by a humanmodel to show the product and its design, but is shown to the consumeras a “flat” image since it is shown to the consumer on the display ofthe computer being used by the consumer. Thus, the actual design of theproduct and how the product looks in real life is often difficult todetermine from those images. This may result in consumers not purchasingthe product which is undesirable.

Another limitation of these typical E-commerce systems is that theproduct available on the website, such as a shirt, cannot be customizedby the consumer with a design on the product. Thus, the consumer cannotsee the customized product with the design and this also may result inconsumers not purchasing the product which is undesirable. Thus, it isdesirable to provide a system and method that provides better models forproducts and it is to this end that the system and method are directed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrates an exemplary implementation of theproduct modeling system;

FIG. 2 illustrates an exemplary implementation of a product modelingmethod;

FIG. 3 illustrates further details of an exemplary implementation of aproduct modeling method;

FIGS. 4A and 4B illustrate further details of the process for marking aproduct in the product modeling method;

FIGS. 5A and 5B illustrate further details of the process for generatingimages of a product in the product modeling method;

FIGS. 6A-6D illustrate further details of the process for preprocessingthe model in the product modeling method;

FIGS. 7A-7C illustrate further details of the post processing process inthe product modeling method;

FIG. 8A illustrates an example of a design to be placed on a piece ofapparel;

FIG. 8B illustrates a typical image of a piece of apparel with thedesign in a typical system;

FIGS. 8C-8D illustrate the design on a piece of apparel in the productmodeling system;

FIG. 8E illustrates the process for placing the design on the model; and

FIGS. 9A-9C illustrate a process for changing the background againstwhich the piece of apparel with the design is displayed in the productmodeling system.

DETAILED DESCRIPTION OF ONE OR MORE EMBODIMENTS

The system and method are particularly applicable to a system and methodfor modeling a shirt implemented in software on a computer and it is inthis context that the system and method is illustrated and described. Itwill be appreciated, however, that the system and method can be used forvarious products wherein the products may include other apparel andother products in which it is desirable to provide betters models of theproducts. For example, the system may be used for any type of garment orpiece of apparel, any item that can be worn or used by a human being orpet, such as a hat, backpack, dog sweater, etc. and/or any other productin which it is desirable to be able to display the product on a model.In addition, the system may be used with any product in which it isdesirable to be able to display the product (with an irregular surface)with a design on it, such as a skateboard, a shoe. In addition, thesystem may be used to display a design on any item with an irregularsurface, such as a wall, automobile body, a pencil and the like.Furthermore, the system may be used to identify a product/item in avideo wherein a design can be inserted into the product/item in thevideo. In addition, the system and method can be implemented in software(shown in the illustrated implementation), hardware or a combination ofhardware and software and may also be implemented on stand alonecomputing device (shown in the illustrated implementation), a webserver, a terminal, a peer to peer system and the like so that thesystem and method are not limited to the particular implementation ofthe system or method.

FIG. 1 is a block diagram illustrates an exemplary implementation of theproduct modeling system 100. In this implementation, the system isimplemented on a stand alone computing device, such as a personalcomputer and the product modeling system is implemented as one or morepieces of software comprising a plurality of lines of computer code thatare executed by a processing unit in the computing device to implementthe product modeling system. The product modeling system, however, canalso be implemented on other computing devices and computing systems,such as a networked computing system, a client/server system, apeer-to-peer system, an ASP model type system, a laptop computer, amobile device, a mobile cellular phone or any other computing devicewith sufficient processing power, memory and connectivity to implementthe product modeling system and method as described below.

The exemplary implementation of the system may include a display device102 to permit a consumer to view the product with the design generatedby the product modeling system, a chassis 104 and one or moreinput/output devices 105, such as a keyboard and mouse, that allow theconsumer to interface with the computing device and the product modelingsystem. The chassis 104 may house a processing unit 106 such as an Intelprocessor, a persistent storage device 108 such as a hard disk drive anda memory 110 wherein the memory may store the softwaremodules/applications being executed by the processing unit. When theproduct modeling system is being implemented on the computing device,the computing device may also include a product modeling store 112, suchas a software implemented database and the memory may store an operatingsystem 114 that controls the operations of the computing device and aproduct modeling module 116 that has a plurality of lines of computercode wherein the plurality of lines of computer code are executed by theprocessing unit to implement the product modeling system and method asdescribed below.

For purposes of illustrating the product modeling system and method, aproduct modeling method for a piece of apparel, such as a t-shirt, witha design is described below. However, the product modeling system mayalso be used for other products, such as other apparel and otherproducts in which it is desirable to provide betters models of theproducts. For example, the system may be used for any type of garment orpiece of apparel, any item that can be worn or used by a human being orpet, such as a hat, backpack, dog sweater, etc. and/or any other productin which it is desirable to be able to display the product on a model.In addition, the system may be used with any product in which it isdesirable to be able to display the product (with an irregular surface)with a design on it, such as a skateboard, a shoe. In addition, thesystem may be used to display a design on any item with an irregularsurface, such as a wall, automobile body, a pencil and the like.Furthermore, the system may be used to identify a product/item in avideo wherein a design can be inserted into the product/item in thevideo. The output of the product modeling method (an image of theproduct with a design shown on the product) may be used for variouspurposes. For example, the output may be used to generate a plurality ofproduct displays with designs on a website that allows consumers to seethe products. The example described below is a system in which theproduct modeling system is tied to a product marketing and sellingcompany wherein the product marketing and selling company has control ofmodels and images of the product modeling system. In anotherimplementation/embodiment of the product modeling system, the system maypermit a consumer to provide their own images/models, such as models ofthe actual consumer, so that the consumer can upload the image to aservice and then have the selected design displayed on the model of theactual consumer wherein the service provides: 1) the model components(to create the model form); 2) a tool to upload/modify the model imagesto the service; and 3) a tool to display the model with the design tothe consumer.

FIG. 2 illustrates an exemplary implementation of a product modelingmethod 120 that displays a model with a design on the model wherein themodel is a realistic representation of a person with a piece of apparelthat has the design on the piece of apparel. The methods shown in FIGS.2 and 3, the processes described below may be performed by the productmodeling module 116 described above. A consumer may select a design(122) such as the design shown in FIG. 8A and a warp process (124) maybe performed to generate a warp design (128). The consumer may alsoselect a background (126) for the model such as the backgrounds shown inFIGS. 9A-9C. Once the background and design are chosen by the consumer,the design is warped and then surface shading (130) and a surfacespecular process (132) is performed. Once these processes are completed,the model is created with the design (134) wherein the model with thedesign is shown to the consumer. In one embodiment, the model with thedesign is displayed to the consumer to assist the consumer in previewingthe product with the design before the consumer purchases the productwith the design, such as through an E-commerce website. Now, the productmodeling method is described in more detail.

FIG. 3 illustrates further details of an exemplary implementation of aproduct modeling method 140 when used with a piece of apparel. Theproduct modeling method (and the processes set forth below) areimplemented, in one embodiment and implementation, as a plurality oflines of computer code that are part of the product modeling module thatare executed by a processing unit 106 that is part of the productmodeling system. In the method, a piece of apparel is created with aplurality of markers (142) that are used to capture information aboutthe piece of apparel when the piece of apparel is worn by a human model.The plurality of markers may be a marker pattern that encodes, in twodimensions, a flexible substrate that may be detected when the flexiblesubstrate is placed on a complex three dimensional surface wherein thecoverage area of the marker pattern does not substantially occlude thesubstrate that it encodes. For example, the plurality of markers maycover a predetermined percentage, such as 50%, of the piece of apparel,that allow the system to capture information about the piece of apparelwhen the piece of apparel is worn by a human model. In oneimplementation, the plurality of markers may form a grid. In moredetail, the markers that form a grid on a flat surface (the piece ofapparel flat on a surface when the markers are properly positioned onthe piece of apparel) may be used to map to a grid of markers on anon-flat surface (the piece of apparel when worn on a human model). Asshown in FIG. 4A, the grid of markers 186 on the flat surface are mappedto a grid 187 with the same markers in the same positions on a non-flatsurface so that the mapping between the grid on the flat surface and thegrid on the non-flat surface is determined. The system may interpolatethe marker locations to generate a mapping from the plurality of markersto the grid on the flat surface and may then store the mapping to avoidrecalculation of the mapping each time. In one embodiment, the markersmay be a number of non-visible lines that form a grid. In anotherembodiment, the markers may be a plurality of optical markers 190 thatmay be affixed to a piece of apparel 192 as shown in FIG. 4B thatpermits the optical tagging of the piece of apparel to map the surfaceof the piece of apparel when worn by a human model. The optical markersmay be made of a reflective material, a colorized material or adiffraction pattern. The reflective material may be retro-reflectivematerial. The colorized material may be pigmented material. The markersmay have various shapes (including the dot shape shown in FIG. 4B) andsizes and the method is not limited to any particular shape of themarkers. In one embodiment, the plurality of markers may be a filmmaterial that has the retro-reflective material in a particular shape.In yet another embodiment, the markers may be a set of markers that forma grid wherein the markers are placed onto the piece of apparelelectronically or by other means. In one embodiment in which the productmodeling system is used by a business entity that sells apparel, eachpiece of apparel is placed onto a plurality of human models of differentshapes and/or sizes (as shown in FIGS. 5A and 5B) so that the consumercan then choose a model for the piece of apparel that is closest to theintended wearer of the piece of apparel. In another embodiment in whicheach consumer may create his own model for a piece of apparel, theconsumer is provided with the markers (either electronically or asphysical markers) so that the consumer can affix the markers to a pieceof apparel and then performs the other processes described below. In yetanother embodiment, the product modeling system may allow a plurality ofusers (such as a community of users) to generate a plurality of modelsthat may then be uploaded to the product modeling system.

Once the one or more pieces of apparel are prepared with the markers, animage for each piece of apparel on each different human model may begenerated (150) such as by using a camera to take a picture of the pieceof apparel being worn by a human model. Prior to taking the image of thepiece of apparel with the markers on the human model, the lighting fortaking the image is determined. When the user/consumer generates themodels, the product modeling system may provide instructions for takingan image of the piece of apparel such as using a flash, using aparticular exposure, etc. . . . In one implementation of the productmodeling system, the product modeling system may download a piece ofcode directly to a user/consumer's camera, such as a digital camera, toset up the camera properly to take the image of the product or item. Inparticular, the surface model and illumination model for each piece ofapparel is determined which also allows the color and lighting for theimage to be accurately determined.

Once the image of the piece of apparel on a plurality of human models ina plurality of different poses are taken, the model for the piece ofapparel on a particular model in a particular pose are preprocessed(160) by the product modeling system. During the preprocessing, theproduct modeling system may detect the plurality of markers on the pieceof apparel image, remove the marker images from the image of the pieceof apparel and then generate a representation of the surface of thepiece of apparel when worn by the human model.

In one implementation, the markers may be detected by a distinguishingfeature of the markers (spectral difference, reflective difference,textual difference and/or temporal difference), refined by matchinggeometric properties of the pattern (local pattern finding) andreconstructed by matching the known pattern (local patterns assembledinto a known complete pattern.) The reconstructed pattern may then beused to model the shape of the flexible substrate. The product modelingsystem may have a plurality of local samples of the original unmarkedsubstrate so that the marker pattern can be replaced using the texturesof the unmarked substrate as an example that yields an unmarked imagesuitable for commercial display.

The preprocessing process is shown in FIGS. 6A-6C with FIG. 6Aillustrating the image of the piece of apparel with the markers, FIG. 6Billustrating the plurality of markers identified on the piece of appareland FIG. 6C illustrates the image of the piece of apparel with themarkers removed. Appendix A (2 pages), incorporated herein by reference,contains an example of the pseudocode for identifying the markers on aproduct in one implementation of the product modeling system. The stepsof the marker identification process for one implementation are setforth in Appendix A. In one implementation, the markers are detected byvisible detection. In another implementation of the system, the markersmay be detected by a temporal process in which infrared radiation may beused to image the markers at several different times and then thepattern of the markers is detected based on the images of the markers atseveral different times.

During the identification of the markers, the product modeling systemmay use various techniques. For example, edge detection may be used toidentify each marker and the spacing between the markers that can thenbe used to generate the grid of markers on the surface of the piece ofapparel when worn on a human model that thus allows the surface of thatpiece of apparel on the particular human model in a particular pose tobe accurately determined. Alternatively, the system may threshold at thewhite color based on the color calibration and then locate elementsabove the threshold and then also identify the background includingelements of the human model such as jewelry, an eye or the backgroundbehind the human model. The system may also use histograms to identifythe markers and the background.

The marker images (once identified) may be removed from the image of thepiece of apparel (as shown in FIG. 6C) by various processes. Forexample, the markers may be removed by, for each marker location,identifying the texture adjacent the marker and then filling in thelocation of the marker with the texture in the adjacent area.Alternatively, the system may use image coherence and synthesize theimage to remove the markers in the image.

To generate the representation of the contours of the surface of thepiece of apparel when worn by a particular human model in a particularpose, the system maps the position of the markers 190 relative to eachother as shown in FIG. 6D into a set of contour curves 194 thatrepresent the surface of the piece of apparel when worn by a particularhuman model in a particular pose. Since the system has information aboutthe markers and the grid that they form on a flat surface as shown FIG.4A, the system is able to determine the contours of the surface of thepiece of apparel when worn by a particular human model in a particularpose.

Once the contours of the surface is determined and the preprocessing iscompleted, the model of the piece of apparel when worn by a particularhuman model in a particular pose may be retouched (162) as needed. Then,the model is post-processed (170) by the product model system. Duringthe post-processing process, the product model system colorizes themodel using a color mapping module that is part of the product modelsystem. The colorizing allows each model for each piece of apparel on aparticular human model in a particular pose to have the piece of apparelconverted into any colors such as the two different colors shown inFIGS. 7A and 7B. As shown in FIG. 7C, the system may use the colorcalibration card with a known spectral response for each session tocalibrate images for the same session. To change the color for the pieceof apparel, the fabric may be wrapped onto a sphere as shown in FIG. 7Cwhich is then mapped to the model to change the color of the model.

Appendices B-E, incorporated herein by reference, illustrate, for aparticular implementation of the product modeling system, the codefor 1) remapping the color of an image using normalized ordinal colordistribution; 2) building a color map in normalized histogram order withan index from a matching color space; 3) building a look-up table toremap the colors from a source sphere to a destination sphere; and 4)remapping the color of the source image with a source sphere color mapto a destination image with the color map of the sphere color object.Using the code set forth in these appendices (and the process stepsdescribed in these appendices), the color mapping process: 1) builds acolor map (the BuildMap code in Appendix C) for the source image using asphere to build a histogram and then a sorted table; 2) builds a remaptable (the BuildReMap table code in Appendix D); 3) remaps the imagecolors (the code in Appendices B and E) onto the product. The system mayalso layer color and texture so that the colorized model of theparticular piece of apparel on the particular human model in theparticular pose more accurately emulates different fabrics and/orthreads of the fabric which results, for example, in an accurateemulation of the printed ink of the design on the piece of apparel withthe particular type of fabric.

Once the colorization is completed, the model for a particular piece ofapparel on a particular human model in a particular pose is integratedinto a service (180) such as a website that has the pieces of apparelwith particular designs for sale to consumers.

When the model is integrated into the service, the product modelingsystem may perform warp mapping (182) on a design selected by theconsumer and permit the user to select a particular background (184). Anexample design is shown in FIG. 8A. The exemplary design shown on apiece of apparel in a typical system with a flat image is shown in FIG.8B. Using the product modeling system, a mapping between the designimage and the surface contour of the model of the particular piece ofapparel on the particular human model in the particular pose (See forexample FIG. 8D) is done so that the design is shown on the model asshown in FIG. 8E is a more realistic three dimensional manner.

During the warp mapping (that may be a bicubic image warp), a grid ofthe design 200 is mapped to the surface contour grid 202 which is thenplaced onto the piece of apparel to generate the more realistic modelfor the piece of apparel with the design as shown in FIG. 8D. In themapping process, a point in the design is mapped to the surface contourgrid which is in turn mapped onto the piece of apparel. The imagebackground can be easily exchanged by the product modeling system asshown in FIGS. 9A-9C.

Although the example provided herein is for a piece of apparel (a shirt)worn by a human being, the product modeling system may be used forvarious different products (other pieces of apparel, other garments,hats, shoes, pet clothing, inanimate objects such as cups) with variousdifferent models (human models, animal models, inanimate models such asrobots or mannequins) and with any number of different poses for themodels since the above example is merely illustrative.

While the foregoing has been with reference to a particular embodimentof the invention, it will be appreciated by those skilled in the artthat changes in this embodiment may be made without departing from theprinciples and spirit of the invention, the scope of which is defined bythe appended claims.

The invention claimed is:
 1. An apparatus for modeling a product,comprising: an imaging device that is capable of taking a single imageof the product on an object and a plurality of markers on the productthat do not occlude a surface of the product; and a computing devicethat identifies a location of each marker and maps the location of eachmarker relative to the other markers using the single image to capture aset of contour curves of a design area when that design area is on theproduct on the object, electronically applies a user design to theproduct without using markers associated with the user design by mappinga user design grid to the set of contour curves of the design area andgenerates a visual representation of the user design on the product whenon the object using the captured lighting, texture and contours of theproduct and the object so that the visual representation of the designon the product has the contour of the surface of the product.
 2. Theapparatus of claim 1, wherein the imaging device detects the pluralityof markers by spectral difference and the coverage of the plurality ofmarkers is determined by spectral difference.
 3. The apparatus of claim1, wherein the computing device recovers the texture of the product froman adjacent area based on spectral difference.
 4. A method for productmodeling, comprising: providing, using an imaging device, a single imageof a product on an object using a plurality of markers that form amarker pattern on the product that does not occlude a surface of theproduct; capturing a contour of a design area when that design area ison the product that is represented on the object in the single image,wherein capturing the contour further comprises identifying a locationof each marker and mapping the location of each marker relative to theother markers using the single image to capture a set of contour curvesof the design area; electronically applying, using a computer, a userdesign to the product without using markers associated with the userdesign and by mapping of a user design grid to the set of contour curvesof the design area; and generating, using the computer, a visualrepresentation of the design on the product when on the object using thecaptured lighting, texture and contours of the product and the object sothat the visual representation of the design on the product has thecontour of the surface of the product.
 5. The method of claim 4 furthercomprising detecting, by the imaging device, the plurality of markersand the coverage of the plurality of markers by spectral difference. 6.The method of claim 4 further comprising recovering, by the computer, atexture and a surface shading of the product from an adjacent area basedon spectral difference.